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Abstract 

Cost  growth  is  a  problem  in  U.S.  Department  of  Defense  (DoD)  aequisitions.  A 
partieular  eomponent  of  eost  growth  is  a  eost  overrun  or  Over  Target  Baseline  (0TB).  In 
2009,  Trahan  found  that  the  Gompertz  growth  eurve  better  predicted  program  Estimates 
at  Completion  (EAC)  for  0TB  contracts.  In  2010,  Thickstun  studied  “the  relationships 
between  overruns  and  a  variety  of  factors,”  but  found  0TB  occurrences  “random”  and 
questioned  the  benefit  of  the  0TB  process  (Thickstun,  2010).  In  this  research,  we  study 
OTB’s  ability  to  effect  improved  program  cost  performance;  we  examine  OTB’s  effect 
on  the  cumulative  Cost  Performance  Index  (CPI)  slope  after  an  0TB  intervention.  We 
find  there  is  no  statistically  significant  change  in  cumulative  CPI  slope  after  0TB.  Eor 
the  data  studied,  an  0TB  investment  does  not  significantly  improve  management’s  ability 
to  earn  cost  value  as  reflected  in  the  cumulative  CPI  slope. 


IV 


/  dedicate  the  completion  of  this  effort  to  Jesus  Christ,  my  wife  and  my  children.  I  love, 
because  He  first  loved  me  (1  John  4:19).  Amen. 


V 


Acknowledgements 


To  my  wife  and  children,  thank  you  for  enduring  this  struggle  with  me;  nothing 
can  separate  us  from  the  love  of  God  that  is  in  Christ  Jesus  our  Lord,  nothing.  I'm 
excited  about  the  next  chapter  in  our  lives;  Lord,  your  will  be  done.  Amen. 

A  special  thanks  to  each  of  my  committee  members.  To  Lt  Col  Unger,  thank  you 
for  teaching  this  "budget  guy"  the  cost  discipline  (I  have  a  new  found  respect  for  the 
difficulty  involved  in  producing  a  solid  cost  estimate)  and  most  importantly,  for 
providing  my  family  and  I  top  cover  during  difficult  times.  To  Dr.  White,  thank  you  for 
the  professional  and  personal  encouragement  throughout  the  thesis  process.  I  know  that  I 
could  not  have  done  it  without  you;  on  many  different  levels,  I  appreciate  your  help  very 
deeply.  To  Lt  Col  Wirthlin,  thank  you  for  taking  a  program  management  interest  in  this 
research  and  thank  you  for  giving  me  the  big  picture  of  the  Defense  Acquisition  System. 

Thanks  to  Kym  Henderson,  Vice  President  Research  and  Standards,  PMI-CPM, 
for  sponsoring  this  effort.  Your  interest  in  my  topic  motivated  me  from  start  to  finish. 

GF A/GCA  classmates;  thanks  for  your  help... I  value  our  friendships  and  I  look 
forward  to  serving  with  you  in  the  future.  To  Col(r)  Edwards,  thank  you  for  mentoring 
me. ..our  prayerful  discussions  have  been  invaluable.  To  the  AFIT  Library  staff;  thank 
you  for  your  patient  support.. .a  special  thanks  to  Mr.  Pat  Colucci. 

A  final  thanks  to  Capt  Lemke;  your  faith  and  fellowship  inspired  me  throughout 
these  last  two  years.  Without  your  dedicated  data  support,  I  would  not  have  finished;  I 
greatly  appreciate  you  and  your  family. 

Dennis  E.  Jack 


VI 


Table  of  Contents 


Page 

Abstract . iv 

Acknowledgements . vi 

Table  of  Contents . vii 

List  of  Figures . ix 

List  of  Tables . x 

I:  Introduction . 1 

Background . 1 

Purpose . 4 

Research  Questions  and  Methodology . 4 

Chapter  Summary . 5 

II;  Literature  Review . 6 

Earned  Value  Management  (EVM)  Overview . 6 

The  Cost  Performance  Index  (CPI) . 11 

Historical  EVM  Research . 13 

Over  Target  Baseline  (0TB)  Overview . 17 

0TB  and  CPI  Interaction . 19 

Chapter  Summary . 21 

Chapter  III:  Data  and  Methodology . 23 

Data  Source . 23 

Data  Exclusions . 23 

Slope  Calculation . 26 

Hypothesis  Test . 27 

Mann- Whitney  U-Test . 30 

Chapter  Summary . 32 

Chapter  IV;  Results  and  Analysis . 33 

Results . 33 

Analysis . 34 

Eimitations . 35 

Chapter  Summary . 36 

Chapter  V:  Discussion  and  Conclusion . 37 

vii 


Page 

Research  Questions,  Results  and  Limitations . 37 

Policy  Implications . 38 

Future  Research . 39 

Appendix  A:  Earned  Value  Managment  'Gold  Card'  (2009) . 41 

References . 42 


viii 


List  of  Figures 


Page 
. 8 


Figure  1:  DAU  Gold  Card,  2009 . 

Figure  2:  Charaeterization  of  the  Cost  Performanee  Index  (CPI) . 12 

Figure  3:  Over  Target  Baseline  (DCMA,  2006) . 17 

Figure  4:  0TB  Proeess  Flow  (DAU,  2007) . 18 

Figure  5:  CPI  Trend  Adjusted/Unadjusted  for  Level  Post  0TB  (Equal  Slopes) . 19 

Figure  6:  CPI  Trend  Adjusted/Unadjusted  for  Level  Post  0TB  (Positive  Slope  Chg.) ...  20 

Figure  7:  Normality  Test  for  Cumulative  CPI  Slopes  Before  0TB . 29 

Figure  8:  Normality  Test  for  Cumulative  CPI  Slopes  After  0TB . 29 

Figure  9:  Similar  Distribution  Shapes;  Cumulative  CPI  Slopes  Before,  After  0TB . 3 1 


IX 


List  of  Tables 


Page 


Table  1:  Analysis  of  DOD  Major  Defense  Aequisition  Programs  (GAO-08-604CG) . 1 

Table  2:  EVM  Performanee  Indiees  (Thiekstun,  2010) . 9 

Table  3:  Cost  Performanee  Data  for  A- 12  Program  (Christensen,  1999) . 10 

Table  4:  A  Range  of  Estimates  at  Completion  for  A-12  Program  (Christensen,  1999)...  10 

Table  5:  0TB  Data  Exelusion  Aoeounting . 26 

Table  6:  Results  of  OTB's  Effeet  on  Cumulative  CPI  Slope  After  0TB . 34 


X 


CONTRACT  OVER  TARGET  BASELINE  (OTB)  EFFECT  ON  EARNED  VALUE 
MANAGEMENT’S  COST  PERFORMANCE  INDEX  (CPI) 


I:  Introduction 

Background 

The  U.S.  Department  of  Defense  and  our  Nation  faee  a  timeless  ehallenge;  mateh 
finite  financial  resources  to  prioritized  joint-capability  requirements  while  earning  the 
greatest  capability  value  per  acquisition  dollar  spent.  The  U.S.  Government 
Accountability  Office  (GAO)  has  rated  the  Department  of  Defense’s  (DoD)  Defense 
Acquisition  System  (DAS)  as  a  “high-risk”  area  since  1990  for  its  costly  and 
“fragmented”  approach  to  identifying  and  acquiring  materiel  solutions  to  meet  joint 
defense  capability  requirements  (GAO,  2009). 

The  inability  to  acquire  joint  defense  capabilities  at  contracted  costs  and  within 
scheduled  timeframes  is  a  continuing  DoD  problem.  As  reported  in  fiscal  year  2008 
dollars  (Table  1),  the  DOD’s  estimated  total  acquisition  cost  growth  relating  to  its 
investment  in  95  major  defense  programs  is  $295  billion;  this  cost  growth  is  accompanied 
by  an  average  schedule  delay  in  delivering  initial  capabilities  of  21  months  (GAO,  2008). 


Table  1:  Analysis  of  DOD  Major  Defense  Acquisition  Programs  (GAO-08-604CG) 


Analysis  of  DOD  Major  Defense  Acquisition  Program  Portfolio  (fiscal  year  2008  dollars) 

Portfolio  status 

Fiscal  year  2007  portfolio 

Number  of  programs 

95 

Change  in  total  research  and  development  costs  from  first  estimate 

40  percent  increase 

Change  in  total  acquisition  cost  from  first  estimate 

26  percent  increase 

Estimated  total  acquisition  cost  growth  from  first  estimate 

$295  billion 

Share  of  programs  with  25  percent  or  more  increase  in  program 
acquisition  unit  cost 

44  percent 

Average  schedule  delay  in  delivering  initial  capabilities 

21  months 

Source:  GAO  analysis  of  DOD  data. 
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A  particular  subcomponent  of  cost  growth  is  a  cost  overrun  or  Over  Target 
Baseline  (0TB)  contraet;  essentially,  an  0TB  refleets  a  eontraetor’s  inability  to  produce  a 
required  capability  at  a  specified  contraeted  eost.  Further,  0TB  signifies  management’s 
deeision  to  establish  a  new  Earned  Value  Management  (EVM)  Performanee 
Measurement  Baseline  (PMB)  “to  improve  managerial  control  over  the  exeeution  of  the 
remaining  work  in  a  projeet”  (DAEl,  2007).  0TB  is  a  very  detailed,  ten-step  process  that 
requires  aetive  commitment  Ifom  all  aequisition  stakeholders  assigned  to  that  eontract 
effort.  Per  the  DAEl  guidebook,  the  0TB  motto  is  “Do  it  onee!  Do  it  right!”  (DAU, 
2007). 

In  2009,  Trahan  found  that  nonlinear  growth  modeling,  speeifically  the  Gompertz 
growth  curve,  better  predieted  program  Estimates  at  Completion  (EAC)  for  0TB 
eontraets  (Trahan,  2009).  In  2010,  Thiekstun  attempted  to  eomplement  Trahan’s 
researeh  by  producing  an  0TB  prediction  model  based  on  logistie  regression  and  found 
that  0TB  is  a  random  oceurrenee  for  the  data  studied  and  questioned  the  value  of  the 
0TB  proeess  (Thiekstun,  2010).  Thiekstun  reports  that  “there  have  been  over  $17  billion 
in  eost  overruns  related  to  OTBs  sinee  2000”  and  for  the  dataset  studied,  “approximately 
twenty  pereent  of  all  aequisition  contracts  in  the  DoD  experienced  eost  overruns  over  the 
past  20  years”  (Thiekstun,  2010). 

At  the  contract  level,  internal  eontrol  aimed  at  produeing  program-level  deeision 
support  was  instituted  by  the  DoD  deeades  ago.  Sinee  the  1960’s,  the  DoD  has  required 
major  defense  acquisition  contractors  to  comply  with  Earned  Value  Management  (EVM) 
standards  and  finaneial  reporting  as  a  means  to  eontrol  cost,  schedule  and  performanee 
(Eleming,  2000).  The  Cost  Performanee  Index  (CPI)  is  a  critical  EVM  cost  performance 
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metric.  The  CPI  is  not  only  a  measure  of  cost  performance  health,  it  is  a  statistic  utilized 
in  predicting  a  program's  Estimate  at  Completion  (EAC).  Eor  its  importance,  the  CPI  has 
been  a  critical  topic  of  academic  research  centered  on  EAC  methods  and  CPI  heuristics. 

In  2008,  Henderson  states  “the  widely  reported  CPI  stability  rule  cannot  be  generalized 
even  within  the  US  Defense  Department  (US  DoD)  project  portfolio”  and  referring  to  the 
goal  of  improving  project  performance,  “an  understanding  of  project  characteristics, 
which  result  in  progressively  improving  CPI  would,  if  these  characteristics  could  be 
emulated  in  other  programs,  be  an  extremely  useful  advance  to  practice”  (Henderson, 
2008).  Identifying  specific  PM  actions  that  improve  the  cumulative  CPI,  correspondingly 
increasing  the  value  of  the  cumulative  CPI  slope,  is  the  goal  of  our  research. 

As  it  relates  to  the  CPI,  an  0TB  intervention  typically  removes  any  cost  variance 
associated  with  contract  performance  and  resets  the  CPI  to  a  value  of  one.  Given  the  CPI 
is  reset  to  one,  the  only  method  of  determining  0TB  effectiveness  on  the  CPI  is  to  study 
the  cumulative  CPI  trend  or  the  cumulative  CPI  slope  after  0TB.  The  cumulative  CPI 
slope  change  (comparing  slopes  before  and  after  0TB)  provides  a  generic  measurement 
that  can  be  examined  for  all  types  of  programs  regardless  of  technical  risk,  appropriation 
and  programmatic  content.  The  goal  of  0TB  is  to  improve  managerial  control  of  a 
project's  remaining  work  in  terms  of  cost,  schedule  and  performance;  if  effective,  0TB 
should  increase  the  cumulative  CPI  slope. 
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Purpose 


Given  “the  DoD  is  entrusted  with  more  taxpayer  dollars  than  any  other  federal 
agency",  it  is  incumbent  upon  management  (at  the  enterprise  and  program  levels)  to 
identify  and  implement  management  actions  that  produce  improved  acquisition  outcomes 
(GAO,  2009).  Our  research  attempts  to  identify  program  management  actions  that 
produce  a  positive  managerial  cost  effect;  specifically,  we  examine  0TB  process  actions 
(treatment)  for  a  positive  effect  on  the  rate  of  earning  cost  value  as  measured  by  the 
cumulative  CPI  slope  after  0TB. 

Research  Questions  and  Methodology 

Our  research  aims  to  answer  the  following  questions: 

1)  Does  the  0TB  process  (treatment)  improve  the  cumulative  CPI's  rate  of 
change  (cumulative  CPI  slope)  after  0TB? 

2)  Is  the  cumulative  CPI  slope  after  0TB  sensitive  to  time  and/or  programmatic 
factors  to  include  contract  type,  military  service  and  the  purpose  of  the  appropriation? 

To  answer  these  questions,  we  examine  the  cumulative  CPI  rate  of  change,  the 
slope  of  the  line  created  by  cumulative  CPI  data  points  before  and  after  0TB.  Similar  to 
past  0TB  and  EVM  research,  the  DoD  Defense  Acquisition  Executive  Summary  (DAES) 
database  is  the  source  of  our  data.  Cumulative  CPI  is  not  distributed  normally,  therefore, 
we  utilize  a  nonparametric  test  (Mann- Whitney  Test)  to  determine  whether  there  is  a 
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statistically  significant  difference  in  the  eumulative  CPI  slopes  before  and  after  0TB, 
whieh  we  deseribe  in  Chapter  3 . 

Chapter  Summary 

0TB  seeks  to  gain  managerial  eontrol  over  remaining  work  in  terms  of  eost, 
sehedule  and  performanee.  Earning  greater  eapability  value  for  every  dollar  spent  is  a 
timeless  ehallenge  that  cuts  across  every  DoD  aequisition  program  regardless  of  life  cyele 
stage,  platform  and  program  risk.  As  stewards  of  taxpayer  funds,  the  DoD  should  exploit 
aequisition  actions  that  produce  improved  eost,  sehedule  and  performanee  outeomes  and 
divest  itself  of  aequisition  notions  that  fail  to  produoe  the  same.  Chapter  II  provides  a 
review  of  past  research  concerning  EVM  and  0TB.  In  Chapter  III,  we  explain  the  souroe 
of  our  data,  present  the  hypothesis  test  and  explain  the  statistioal  test.  In  Chapters  IV  and 
V,  we  summarize  the  results  of  our  analysis  and  provide  policy  implications  based  on  our 
findings. 
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II:  Literature  Review 


In  this  chapter,  we  expand  on  the  EVM  architeeture,  the  historical  body  of  EVM 
research  and  emphasize  CPI  eharacteristies.  Additionally,  we  define  the  term  0TB, 
diseuss  the  0TB  process  and  emphasize  the  stated  purpose  of  0TB. 

Earned  Value  Management  (EVM)  Overview 

Sinee  its  ineeption,  EVM  has  been  a  program  management  (PM)  tool  that  ties 
eost,  schedule  and  performanee  into  an  integrated  program  baseline;  essentially,  the  EVM 
eonstruct  serves  as  a  roadmap  of  exeeution  and  an  internal  eontrol  meehanism  to  assess 
projeet  status  and  future  eompletion.  In  a  memorandum  dated  3  Jul  07,  USD  AT&E 
Kenneth  J.  Krieg  deseribed  EVM  as  a  projeet  management  best  praetiee  that  “provides  a 
disciplined  approaeh  to  managing  projects  successfully  through  the  use  of  an  integrated 
system  to  plan  and  eontrol  authorized  work  to  aehieve  eost,  sehedule  and  performanee 
objectives”  (Krieg,  2007). 

Contraetor  earned  value  management  systems  (EVMS)  rest  on  32  guidelines  or 
industry  standards  established  by  Ameriean  National  Standards  Institute 
(ANSI)/Eleetronie  Industries  Allianee  (EIA);  “the  DoD  formally  adopted  ANSI/EIA-748 
in  August  1998  for  applieation  to  major  defense  aequisition  programs”  (DCMA,  2006). 
The  guidelines  are  not  preseriptive,  but  give  government  eontraetors  the  flexibility  to 
develop  business  information  systems  that  aecurately  eollect  and  report  aequisition 
program  exeeution  data  to  enable  resouree  deeision-making  (DCMA,  2006).  0MB 
Cireular  No.  A-1 1  (0MB  A-1 1),  Seetion  300,  establishes  poliey  for  planning,  budgeting, 
aequisition  and  management  of  Eederal  eapital  assets  “to  ensure  searee  publie  resourees 
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are  wisely  invested”  (0MB,  Jun  08).  0MB  A-1 1  referenees  EVM  contract  criteria  set 
forth  in  the  Federal  Acquisition  Regulation  (FAR).  FAR  7. 105(b)(10)  discusses  FVMS 
performance  analysis  and  calls  for  FVMS  compliance  language  in  written  acquisition 
plans.  Further,  FAR  34.20 land  34.202  mandate  FVMS  for  major  development 
acquisition  in  accordance  with  agency  procedures  and  ANSI/FIA-748  standards  and  calls 
for  program  Integrated  Baseline  Reviews  (IBR)  when  FVMS  is  required.  FAR  34.203 
directs  the  insertion  of  an  EVMS  contract  clause  in  solicitations  requiring  a  contractor 
FVMS.  In  terms  of  agency  procedures,  the  Defense  Federal  Acquisition  Regulation 
Supplement  (DEARS)  234.2  and  the  DoD  Instruction  5000.2,  Operation  of  the  Defense 
Acquisition  System,  prescribe  mandatory  EVM  requirements  for  cost  and  incentive 
contracts.  These  regulations  require  formal  compliance  validation  of  contractor  EVMS 
with  ANSI/EIA-748  for  cost  and  incentive  contracts  >  $50  million.  Further,  for  cost  and 
incentive  contracts  from  $20-50  million,  a  formal  validation  is  not  required,  but 
ANSI/EIA-748  compliance  is  required.  Finally,  for  any  contract  less  than  $20  million, 
PMs  have  discretion  and  can  decide  whether  the  cost  of  an  EVMS  is  justified  by  its 
benefits  (DAU,  2009). 

Beyond  the  purpose  and  regulatory  requirements,  EVM  is  a  simple  and  useful  PM 
tool.  The  foundation  of  EVM  is  the  Performance  Measurement  Baseline  (PMB);  Cukr 
describes  it  well  by  stating  "the  purpose  of  a  performance  measurement  baseline  is  to 
capture  the  technical  work  and  performance  requirements,  the  time  limitations,  and  the 
resource  constraints  of  a  project  in  a  time -phased,  dollarized  plan  for  successfully 
accomplishing  the  project"  (Cukr,  2000).  The  importance  of  an  accurate  and  disciplined 
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PMB  cannot  be  overstated.  The  DAU  EVM  "Gold  Card"  (Appendix  A)  provides  a 
summary  of  EVM  calculations  and  terms  (DAU,  2009). 


Figure  1;  DAU  Gold  Card,  2009 


The  PMB,  the  Budgeted  Cost  of  Work  Scheduled  (BCWS)  or  the  Planned  Value 
(PV)  is  the  starting  point  of  EVM  analysis;  "the  focus  of  earned  value  has  been 
consistent:  the  accurate  measurement  of  physical  performance  against  a  detailed  plan  (or 
PV)  to  allow  for  the  accurate  prediction  of  the  final  costs  and  schedule  results  for  a  given 
project"  (Fleming,  2000).  As  a  program  executes  and  data  is  collected  Ifom  Contract 
Performance  Reports  (CPR),  the  Budgeted  Cost  of  Work  Performed  (BCWP)  or  the 
Earned  Value  (EV)  is  compared  to  the  Actual  Cost  of  Work  Performed  (ACWP)  or  the 
Actual  Cost  (AC)  and  the  Budgeted  Cost  of  Work  Scheduled  (BCWS)  or  the  Planned 
Value  (PV)  at  "time  now"  to  determine  the  Cost  Variance  (CV)  and  the  Schedule 
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Variance  (SV),  respectively.  These  variances,  positive  or  negative,  provide  program 
managers  insight  into  the  current  cost  and  schedule  status  of  the  project.  At  "time  now", 
two  performance  metrics  can  be  generated  by  dividing  EV  by  AC  and  EV  by  PV.  The 
first  calculation,  EV/AC,  produces  the  Cost  Performance  Index  (CPI).  The  CPI  is  a  "time 
now"  metric  that  measures  contractor  cost  performance.  The  second  calculation,  EV/PV, 
produces  the  Schedule  Performance  Index  (SPI).  The  SPI  measures  contractor  schedule 
performance.  EVM  cost  analysts  and  PMs  utilize  the  CPI,  the  SPI  and  various 
combinations  of  the  two  indices  (SCI  product  and  Composite  additive  weighting)  as 
performance  factors  (Table  2)  to  calculate  a  range  of  Estimates  at  Completion  (EAC). 

Table  2:  EVM  Performance  Indices  (Thickstun,  2010) 

CPI=  Budgeted  Cost  of  Work  Peifoiiiied  (BCA\T) 

Actual  Cost  of  Work  Perfomied  (ACWP) 

SPI  =  Budgeted  Cost  of  Work  Pei  foiiiied  (BCWT) 

Budgeted  Cost  of  Work  Scheduled  (BCWS) 

SCI  =  CPI  *  SPI 

_ Composite  Index  -  (wl  *CPI)  +  (w2*SPI) _ 


Once  the  performance  factors  are  determined  and  the  Budget  At  Completion 
(BAC)  is  known,  index-based  EACs  are  calculated  by  adding  the  AC  to  the  quotient  of 
remaining  work  divided  by  a  selected  performance  factor  (Table  2).  Christensen 
provides  an  excellent  example  of  EVM's  simplicity  and  utility  in  calculating  a  range  of 
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EACs.  In  the  wake  of  the  A- 12  eaneellation,  Christensen  developed  Tables  3  and  4  to 
demonstrate  the  utility  of  the  earned  value  management  report  (Christensen,  1999). 


Table  3  (Christensen,  1999) 

Cost  Performance  Data  for  A- 12  Program 
(April  1990,  Millions  of  Dollars) 


Month 

BCWS 

BCWP 

ACWP 

SV 

CV 

BAG 

EAC 

VAC 

April 

2,080 

1,491 

1,950 

(589) 

(459) 

4,046 

4,400 

(354) 

(See  Appendix  A  for  EVM  acronym  definitions  and  equations.) 


Table  4  (Christensen,  1999) 

A  Range  of  Estimates  at  Completion  for  A- 12  Program 
(Derived  Ifomthe  Cumulative  Performance  Data  in  Table  2) 


Performance  factor 

Performance  factor  value 

EAC  (Millions) 

CPI  X  SPl 

0.5481 

$  6,612 

SPI 

0.7168 

$  5,514 

.8  CPI  X  .2  SPI 

0.7551 

$5,334 

CPI 

0.7646 

$  5,292 

(See  Appendix  A  for  EVM  acronym  definitions  and  equations.) 


Having  calculated  the  index-based  performance  factors  (Table  4)  Ifom  the  cost 
data  (Table  3),  Christensen  effectively  displayed  a  lower  and  upper  bound  range  of 
Estimates  at  Completion  (EAC)  available  to  A-12  program  leadership  (Table  4). 
Christensen’s  example  demonstrates  the  relative  simplicity  of  EVM  calculations,  but 
highlights  its  usefulness  in  decision  support. 
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Commenting  on  the  large  number  of  EAC  ealeulation  formulas,  Fleming  and 
Koppelman  eharaeterize  the  eumulative  CPI  based  EAC  ealculations  (Table  4)  as  "three 
of  the  more  aeeepted  formulas"  (Fleming  and  Koppelman,  2000). 

The  Cost  Performance  Index  (CPI) 

Charaeterization  of  the  CPI  is  partieularly  important  to  our  research.  Fleming  and 
Koppelman  characterize  the  CPI  as  a  “delicate  relationship  between  the  value  of  the  work 
physically  completed  and  in  process,  related  to  the  actual  costs  incurred  for  doing  such 
work”  (Fleming  and  Koppelman,  2000).  Additionally,  the  CPI  is  a  generic  metric  that 
accommodates  all  types  of  programs  and  levels  of  technical  risk  and  “reflects  the  health 
of  (a)  project”  (Fleming  and  Koppelman,  2000).  Regardless  of  the  specific  program,  the 
CPI  highlights  cost  variance  and  directs  management  attention  to  negative  trends. 

By  definition,  the  Cost  Performance  Index  is  BCWP/ACWP,  the  quotient  of 
Budgeted  Cost  of  Work  Performed  (BCWP)  divided  by  the  Actual  Cost  of  Work 
Performed  (ACWP);  the  CPI  is  a  measure  of  cost  efficiency  or  cost  performance  relating 
Earned  Value  (EV)  to  Actual  Cost  (AC).  Cumulative  CPI  (CPCum)  relates  total  EV  to 
total  AC  for  “time  now”,  while  CPI  in  general  can  relate  EV  to  AC  for  any  defined 
period.  CPI  values  less  than  one  indicate  an  unfavorable  overrun  condition  and  CPI 
values  greater  than  one  indicate  a  favorable  underrun  condition.  In  a  scenario  of  perfect 
knowledge  and  perfect  execution,  the  CPI  is  consistently  a  value  of  one.  Figure  2  depicts 
all  three  scenarios. 
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CPIcum  Characterization 
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Figure  2:  Characterization  of  the  Cost  Performance  Index  (CPIcum) 


The  CPI  provides  a  great  deal  of  programmatic  insight,  especially  into  the  PMB’s 
technical  risk.  Recall  that  the  PMB  is  a  “time-phased,  dollarized  plan”  that  represents  the 
planned  technical  work  packages  for  a  particular  program;  it  is  an  estimated  plan  (Cukr, 
2000).  With  any  estimated  plan,  there  exists  risk  and  uncertainty  and  that  uncertainty  is 
reflected  in  the  actual  execution  of  the  program.  In  a  “perfect  knowledge”  scenario,  the 
program  produces  a  capability  exactly  as  planned;  the  program  earns  $1  dollar  of  value 
for  every  $1  dollar  of  actual  cost.  The  program’s  CPI  is  perfectly  constant  at  a  value  of 
one  (Figure  2)  and  the  cumulative  CPI  slope  is  horizontal.  Depicted  as  the  Overrun  CPI 
slope,  the  program  that  lacks  perfect  knowledge  and  estimates  optimistically  will  earn 
less  than  $1  dollar  of  value  for  every  $1  dollar  of  cost.  Conversely  and  depicted  as  the 
Underrun  CPI  slope,  the  program  that  lacks  perfect  knowledge  and  estimates 
conservatively  will  earn  greater  than  $1  dollar  of  earned  value  per  $1  dollar  of  cost.  CPI 
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directly  reflects  the  cost  performance  of  the  program  and  provides  insight  into  program 
technical  risk  and/or  the  quality  of  the  estimating  process.  Additionally,  the  slope  of  the 
cumulative  CPI  provides  insight  into  management’s  ability  or  inability  to  actively 
improve  its  cost  position  and  performance. 

Historical  EVM  Research 

EVM  research  centers  on  EAC  prediction  methods  and  cumulative  CPI  heuristics. 
In  a  comprehensive  review  of  twenty- five  proposed  or  comparative  EAC  studies  that 
explored  index,  regression  and  other  methods,  Christensen  concluded  that  “no  one 
formula  or  model  is  always  best”  and  “the  accuracy  of  index-based  formulas  depends  on 
the  type  of  system,  and  the  stage  and  phase  of  the  contracts”  (Christensen,  1995).  Since 
that  time,  Tracy  examined  regression  based  EAC  models  and  found  that  regression 
models  only  outperform  index-based  models  at  early  stages  of  completion  (Tracy,  2005). 
In  2009  Trahan  produced  three  EAC  models  using  the  Gompertz  growth  curve  and 
concluded  that  growth  models,  depending  on  model  and  phase,  are  “a  more  accurate 
estimating  tool  for  identified  0TB  contract’s  EAC  as  compared  to  the  CPI,  SCI  and 
Composite  Index  methods”  (Trahan,  2009).  Noted  by  Thickstun,  these  findings  add 
further  support  to  Christensen's  1995  research  that  there  is  no  one  EAC  method  that 
outperforms  the  others  in  all  situations  (Thickstun,  2010).  Eurther,  through  logistic 
regression  analysis  of  various  0TB  program  factors,  Thickstun  attempted  to  complement 
Trahan’s  research  by  developing  a  model  to  predict  0TB  contracts.  Thickstun  concluded 
that  “the  ability  to  predict  OTBs  was  no  better  than  a  coin  flip”  for  the  data  studied  and 
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“it  suggests  that  OTBs  may  occur  randomly”;  she  goes  on  to  question  the  benefits  of  the 
0TB  process  (Thickstun,  2010). 

Within  EVM,  the  CPI  is  a  central  index  utilized  in  almost  every  performance 
factor  EAC  calculation.  Additionally,  CPI  research  has  developed  rules  of  thumb 
(heuristics)  that  empower  PMs  to  test  EAC  confidence  and  understand  program  stability. 
The  following  CPI  heuristics  are  particularly  useful  in  evaluating  a  contractor’s  EAC. 

•  “Research  has  shown  that  the  EAC  derived  Ifom  the  CPI  is  a  reasonable  floor  to 
the  final  cost”  (Christensen,  1996). 

•  “When  the  cumulative  CPI  is  significantly  less  than  TCPI,  it  is  highly  doubtful 
that  the  contract  will  be  completed  at  the  EAC”  (Christensen,  1999). 

•  “The  smallest  and  largest  EACs  were  derived  Ifom  the  CPI  and  the  product  of  the 
CPI  and  SPI,  respectively”  (Christensen,  1999). 


The  PM’s  ability  to  improve  cost  performance  is  particularly  important  to  our 
research  and  directly  tied  to  CPI  stability.  Concerning  DoD  cumulative  CPI  stability, 
defined  by  Christensen  as  “cumulative  CPI  does  not  change  by  more  than  plus  or  minus 
0.10  Ifom  its  value  at  the  20  percent  completion  point”,  the  following  heuristics  are  cited. 

•  DoD  research  supports  the  fact  that  DoD  programs  are  unable  to  change  their 
cumulative  CPI  by  +/-  10%  once  the  20%  program  completion  point  is  achieved 
(Christensen  and  Payne,  1992). 

•  “A  stable  CPI  is  evidence  that  the  contractor’s  management  control  systems, 
particularly  the  planning,  budgeting,  and  accounting  systems  are  functioning 
properly”  and  “thus  indicate  that  the  contractor’s  final  costs  of  authorized  work, 
termed  ,fi;stimate  at  Completion,’  are  reliable”  (Christensen  and  Payne,  1992). 

•  “Knowing  that  the  CPI  is  stable  may  help  the  analyst  evaluate  the  capability  of  a 
contractor  to  recover  Ifom  a  cost  overrun  by  comparing  the  CPI  with  other  key 
indicators”  (Christensen  and  Payne,  1992). 
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•  Cumulative  CPI  “does  not  ehange  by  more  than  10  pereent  onee  the  eontraet  is  20 
pereent  eomplete;  in  faet,  it  tends  to  decrease”  (Christensen,  1993). 

•  “Recoveries  from  cost  overruns  on  defense  contracts  are  extremely  rare, 
especially  when  the  project  is  more  than  20  percent  complete”  (Christensen, 

1999). 

•  “Based  on  an  analysis  of  155  defense  acquisition  contracts,  Christensen  and  Heise 
(1993)  reported  that  the  range  of  the  cumulative  CPI  from  the  20  percent 
completion  point  to  contract  completion  was  less  than  0.20  for  every  contract. 

This  result  is  usually  interpreted  to  mean  that  the  cumulative  CPI  does  not  change 
by  more  than  plus  or  minus  0.10  from  its  value  at  the  20  percent  completion  point, 
and  is  used  to  evaluate  the  reasonableness  of  projected  cost  efficiencies  on  future 
work”  (Christensen  and  Templin,  2002). 


In  2008,  Henderson  and  Zwtkael  re-examined  CPI  stability  and  found  contrary 
evidence  inside  and  outside  of  the  DoD.  In  their  study  of  twelve  Israeli  Hi-Tech  projects, 
twenty  United  Kingdom  construction  projects  and  five  Australian  IT  projects  for 
cumulative  CPI  stability,  they  state  "this  research  does  not  support  the  previously 
referenced  generalizations  that  the  CPI  stability  rule  has  universal  applicability  for  all 
projects  utilizing  the  EVM  method"  (Henderson,  2008).  Specifically  concerning  the 
aforementioned  international  contracts,  Henderson  and  Zwikael  found  "that  (CPI) 
stability  is  usually  achieved  very  late  in  the  project  lifecycle,  often  later  than  80% 
complete  for  projects  in  these  samples"  (Henderson,  2008). 

Further,  Henderson  cites  contradictory  evidence  within  the  DoD.  In  1996, 
Michael  Popp,  U.S.  Naval  Air  Systems  Command  (NAVAIR),  conducted  a  study 
concerning  the  confidence  level  of  programs  not  breaching  10%  over  budget.  To  answer 
a  question  posed  by  NAVAIR's  Program  Executive  Officer  (PEO),  Popp  and  staff 
developed  "probability  distributions  of  EAC's  (based  on  Cost  Performance  Index  at 
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Complete  (CPI))  based  on  current  CPI  and  %  complete  of  programs  based  on  history" 
(Popp,  1996).  Simplifying  the  question,  Popp  asked,  "given  a  program  that  has  CPI  of  X 
and  a  percent  complete  of  Y,  what  is  the  most  likely  finishing  CPI"  (Popp,  1996).  Popp's 
charts  display  the  correlation  of  cumulative  CPI  at  a  defined  range  of  percent  complete  to 
final  CPI.  By  90-100%  complete,  the  correlation  is  almost  exactly  one  for  all  programs. 
The  greatest  deviation  (from  the  correlation  value  of  one)  is  seen  in  the  10-20% 
completion  chart.  Using  Popp's  correlation  charts  and  a  -l-/-10%  CPI  stability  enclosure 
technique,  Henderson  concludes  from  Popp's  data  that  "CPI  stability  was  also  achieved 
very  late  in  the  project  lifecycle,  often  as  late  as  70-80%  completion"  and  "this  finding  is 
consistent  with  late  CPI  stability  findings  for  the  (international)  commercial  sector" 
(Henderson,  2008). 

The  purpose  of  our  research  is  not  to  dispute  past  CPI  heuristics,  but  to  inquire 
into  PM  actions  that  produce  an  increase  in  final  cumulative  CPI.  At  the  10-20% 
complete  point,  Popp's  correlation  chart  does  not  display  a  straight  line  value  of  one, 
meaning  DoD  programs  and  program  managers  have  the  ability  to  effect  final  CPI 
change  for  better  or  for  worse.  Our  research  utilizes  the  0TB  construct  to  define  a  set  of 
PM  actions  to  study  for  its  treatment  effect  on  cumulative  CPI. 

In  a  broader  EVM  sense,  Fleming  and  Koppelman  state  that  "final  forecasted 
results  are  not  necessarily  preordained"  and  "final  project  results  can  often  be  altered,  but 
only  when  aggressive  management  actions  are  taken"  (Fleming  and  Koppelman,  2000). 
Consistent  with  EVM  research  and  having  noted  several  important  tradeoff  variables, 
Fleming  and  Koppelman  stress  that  aggressive  action,  "if  taken  early,"  can  change  project 
outcomes  (Fleming  and  Koppelman). 
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Over  Target  Baseline  (OTB)  Overview 

All  of  EVM  starts  with  the  Performance  Measurement  Baseline  (PMB);  the  PMB 
is  an  integrated  cost,  schedule  and  performance  execution  plan.  In  terms  of  changing  the 
PMB,  Cukr  states,  “the  reasons  fall  into  three  major  categories:  authorized  contract 
changes  (negotiated  changes  and  authorized  unpriced  work),  internal  replanning,  and 
inadequate  remaining  budget  in  the  contract  with  a  resulting  requirement  for  an  OTB” 
(Cukr,  2000).  From  a  requirements  and  funding  viewpoint,  authorized  contract  changes 
represent  requirement  growth  with  commensurate  funding  growth,  internal  replanning 
represents  a  reallocation  of  existing  contract  funds  to  existing  contract  requirements  and 
OTB  represents  the  contractor’s  need  for  additional  funds  to  perform  the  unchanged 
contract  budget  base  requirements  (Cukr,  2000). 

In  summary,  OTB  reflects  a  contractor’s  inability  to  produce  a  required  capability 
at  a  specified  contracted  cost  and  signifies  management’s  decision  to  establish  a  new 
EVM  Performance  Measurement  Baseline  (PMB)  “to  improve  managerial  control  over 
the  execution  of  the  remaining  work  in  a  project”  (DAU,  2007).  OTB  (cost  overrun) 
increases  Total  Allocated  Budget  (TAB)  beyond  a  constant  Contract  Budget  Base  (CBB). 


Before  overrun 

Total  Allocated  Budget  (TAB) 

Contract  Budget  Base  (CBB) 

Performance  Measurement  Baseline 
(PMB) 

Management 

Reserve 

After  overrun 

Total  Allocated  Budget  (TAB) 

Contract  Budget  Base  (CBB) 

Over  Target  Budget 

Performance  Measurement  Baseline  (PMB) 

Management 

Reserve 

Figure  3:  Over  Target  Baseline  (DCMA,  2006) 
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The  0TB  process  entails  ten  steps  in  which  the  contractor  and  the  customer  work 
together  to  establish  a  new  bottoms-up  estimate  and  integrated  plan  (PMB)  for  a 
program’s  remaining  work.  The  process  involves  customer  approval,  consensus  on  work 
scope,  consensus  on  the  master  schedule,  a  collaborative  risk  analysis  and  a  detailed 
understanding  of  work  packages  tied  to  budget  and  time  constraints  at  the  Control 
Account  Manager  (CAM)  level.  The  entire  0TB  process  (Figure  4)  aims  at  regaining 
managerial  control  of  remaining  work  in  terms  of  cost,  schedule  and  performance. 


Figure  4;  0TB  Process  Flow  (DAU,  2007) 
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OTB  and  CPI  Interaction 


During  the  OTB  process,  program  managers  have  the  choice  to  remove  all  or 
some  of  the  EVM  variances;  typically,  elimination  of  all  variances,  “is  the  most  common 
form  of  variance  adjustment  in  an  OTB  situation”  (DAU,  2007).  Cukr  states,  “this  action 
(eliminate  variances)  makes  sense  if  you  consider  that  the  OTB  essentially  builds  the  past 
variance  trend  into  the  baseline  through  the  contractor’s  estimate,  upon  which  the  OTB  is 
built”  (Cukr,  2000).  In  detail,  elimination  of  the  cost  variance  means  setting  the 
Budgeted  Cost  of  Work  Performed  (BCWP)  equal  to  the  Actual  Cost  of  Work  Performed 
(ACWP),  which  adjusts  the  cumulative  CPI  value  to  one. 

Adjusting  the  CPI  to  one  is  typical  in  OTB  situations,  but  not  necessary  for  our 
study.  More  generically,  our  research  examines  the  slope  of  the  cumulative  CPI  trend 
line  (typically  a  negative  slope  or  growing  overrun)  and  is  not  necessarily  interested  in 
the  level  of  the  cumulative  CPI  metric.  To  clarify.  Figure  5  depicts  two  CPI  trend  lines 
of  equal  slope  value  post  an  OTB  intervention  (OTB  at  Time  Zero). 
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Figure  5;  CPIcum  Trend  Adjusted/Unadjusted  for  Fevel  Post  OTB  (Equal  Slopes) 
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In  contrast  to  the  equal  pre-  and  post-slopes  in  Figure  5  above  and  consistent  with 
the  theory  of  our  research,  Figure  6  depiets  CPI  adjusted  and  unadjusted  for  level  post 
0TB  with  a  horizontal  eumulative  CPI  slope.  Figure  6  implies  that  the  0TB  proeess  has 
positively  affected  the  eumulative  CPI  slope  and  the  program  in  question  is  now 
“perfectly”  earning  $1  dollar  of  earned  value  for  every  $1  dollar  of  aetual  cost  regardless 
of  CPI  level.  The  theoretical  program,  having  gone  through  the  0TB  process,  has  rightly 
assessed  the  remaining  work,  its  assoeiated  risks  and  has  properly  reprogrammed  a 
commensurate  amount  of  funding  to  the  remaining  effort. 
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Figure  6;  CPIcum  Trend  Adjusted/Unadjusted  for  Level  Post  0TB  (Positive  Slope  Chg.) 
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This  type  of  pre-  and  post-OTB  analysis  is  consistent  with  Cukr’s  direction  to 
0TB  analysts  concerning  the  elimination  of  cost  and  schedule  variances  as  discussed 
above.  Cukr  states  “as  a  result  (of  eliminating  past  variances),  analysts  can  adjust  their 
trend  analysis  by  focusing  on  the  cost  and  schedule  trends  since  the  0TB,  and  comparing 
pre-  and  post-OTB”  (Cukr,  2000).  Analysts  should  allow  "several  months"  of  reporting 
to  occur  prior  to  performing  post  0TB  analysis  based  on  cumulative  indices  (DAU, 
2007).  Consistent  with  this  guidance,  our  analysis  of  post-OTB  cumulative  CPI  slopes 
begins  at  six  months  and  includes  all  cumulative  CPI  points  pre-OTB. 

Chapter  Summary 

In  Chapter  II,  we  reviewed  fundamental  EVM  and  0TB  concepts  to  include  past 
research  and  established  the  relationship  between  EVM  and  0TB.  Specifically,  we 
discussed  OTB's  effect  on  cumulative  CPI  and  characterized  cumulative  CPI  heuristics. 
Historical  DoD  research  supports  the  validity  and  importance  of  index-based  EAC 
calculations  and  displays  the  value  of  CPI  heuristics  in  evaluating  contractor  EACs.  Of 
particular  importance  to  our  current  study  is  cumulative  CPI  stability.  The  vast  majority 
of  DoD  research  (Christensen  et  al)  finds  that  cumulative  CPI  is  stable  at  the  20% 
completion  point,  meaning  that  cumulative  CPI  will  not  deviate  by  +/-  10%  through 
program  completion,  but  recent  research  provides  evidence  against  CPI  stability  with  the 
intent  of  finding  program  actions  that  produce  progressive  cumulative  CPI  improvement 
(Henderson,  2008).  In  Chapter  III,  we  explain  the  source  of  our  data,  provide  the 
cumulative  CPI  slope  calculation,  present  an  0TB  hypothesis  test  that  examines  OTB's 
effect  on  cumulative  CPI  slope  after  0TB  and  explain  the  Mann- Whitney  Test 
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(nonparametric  statistical  test).  In  Chapters  IV  and  V,  we  summarize  the  results  of  our 
analysis  and  provide  policy  implications  based  on  our  findings. 
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Ill:  Data  and  Methodology 

Data  Source 

Consistent  with  previous  EVM  research,  we  utilized  cumulative  and  summary 
EVM  reports  contained  in  the  Defense  Acquisition  Executive  Summary  (DAES) 
database,  retrieved  through  the  Defense  Acquisition  Management  Information  Retrieval 
(DAMIR)  system,  for  all  DoD  Major  Defense  Acquisition  Programs  (MDAP)  as  our 
source  data.  Specifically,  we  compiled  the  cumulative  and  summary  EVM  reports  of  all 
DAES-identified  0TB  contract  efforts  categorized  by  appropriation  purpose  (RDT&E  or 
Procurement),  service  component  (Air  Eorce,  Army,  DoD,  Navy)  and  contract  type  (Cost 
Plus  and  Eixed).  To  increase  data  validity,  we  chose  "system-identified"  0TB  contract 
efforts  (0TB  date  data  field  populated  in  DAMIR)  as  the  focus  of  our  study  to  increase 
assurance  that  the  contractor  and  customer  acknowledged  0TB  status.  Our  decision  to 
utilize  "system-identified"  0TB  contract  efforts  differs  Ifom  Thickstun's  choice  to  use 
DAU's  definition  (TAB  >  CBB)  (Thickstun,  2010);  the  DAES  database  contained 
instances  of  TAB  exceeding  CBB  without  the  0TB  data  field  being  populated.  Given  the 
fact  that  this  study  analyzes  an  0TB  "treatment  effect",  we  chose  the  more  narrow  system 
query  of  0TB  occurrences  to  support  the  assumption  of  contractor  and  customer 
agreement.  Once  compiled,  we  applied  four  data  exclusions  to  arrive  at  our  final  dataset. 
Data  Exclusions 

1 .)  Data  Purification:  The  dataset  contained  duplicate  0TB  dates,  consecutive 
0TB  dates  and  0TB  dates  greater  than  one  per  quarter.  To  adjust,  we  removed  duplicate 
dates  by  sorting  chronologically,  combined  consecutive  dates  to  facilitate  study  and 
removed  0TB  date  occurrences  greater  than  one  per  quarter  to  allow  the  time  necessary 
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to  implement  the  0TB  proeess.  Given  the  number  of  steps  in  the  0TB  proeess  and 
inexaet  time  definition  in  literature,  we  assumed  three  months  as  the  minimum  amount  of 
time  neeessary  to  implement  an  0TB.  DCMA  states  "one  to  two  full  aeeounting  periods 
after  written  authorization  to  proeeed  is  reeeived  should  provide  the  eontraetor  with 
suffieient  time  to  fully  implement  an  OTB/OTS  in  required  reports"  (DCMA,  2006). 
Given  0TB  written  approval  is  required  within  30  days  and  typical  accounting  periods 
are  monthly,  the  assumption  of  three  months  is  consistent  with  the  aforementioned 
exclusion. 

2.)  Unstable  Contract  Budget  Base  (CBB):  Concerning  0TB  implementation,  “it 
is  usually  best  to  isolate  and  separately  implement  the  changes  associated  with 
reprogramming  (0TB)"  (DAU,  2007).  As  discussed  in  our  literature  review,  0TB  is 
"within-scope"  reprogramming,  meaning  that  the  contract  requirement  is  unchanged; 

0TB  reflects  a  contractor's  inability  to  produce  a  defined  requirement  at  a  contracted 
cost,  namely  an  overrun.  To  control  for  requirement  growth  and  to  ensure  we  studied 
similar  requirements  on  either  side  of  0TB  for  treatment  effect,  we  implemented  a  CBB 
stability  rule  based  on  mean  CBB.  We  excluded  all  contract  efforts  that  experienced 
CBB  change  greater  than  +/-10%  of  the  mean  CBB  ((Max  CBB  of  effort  -  Min  CBB  of 
effort)  /  Avg  CBB  of  effort). 

In  terms  of  single-group  research  design,  this  exclusion  increases  internal  validity 
by  removing  an  historical  threat;  if  CBB  has  fluctuated  consistently  throughout  the 
history  of  the  program,  this  fluctuation  of  requirement  contributes  to  a  diminished  effect 
and  threatens  internal  validity  (Trochim,  2008).  Essentially,  it  is  important  to  have  a 
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stable  or  similar  requirement  on  either  side  of  the  0TB  treatment  to  measure  its 
ineremental  effect  on  the  cumulative  CPI  slope. 

3. )  Insufficient  Reports  (Reports  count  <5):  Including  the  0TB  date,  if  an  0TB 
contract  effort  had  less  than  five  reports  (data  points),  it  was  excluded  for  insufficient 
data.  We  were  unable  to  calculate  a  slope  before  and  after  0TB. 

4. )  Multiple  OTBs  Removed:  After  applying  the  previous  exclusions,  only  six 
contract  efforts  contained  two  or  greater  0TB  occurrences.  Given  the  small  sample  size, 
and  low  percentage  of  total  OTBs  studied,  we  excluded  these  OTBs  Ifom  our  study. 
Referencing  the  contractor's  understanding  of  the  overrun  problem  and  the  contractor's 
ability  to  produce  a  valid  plan  for  remaining  work,  DCMA  states  that  multiple  OTBs 
"may  indicate  significant  underlying  management  problems  that  should  be  investigated" 
(DCMA,  2006).  Knowing  that  second  OTBs  are  problematic,  including  this  data  in  our 
study  would  skew  our  results. 

Table  5  accounts  for  our  data  exclusions.  Our  final  dataset  contains  40  contracts, 
with  47  contractual  efforts  having  47  "system- identified"  first  OTBs  (0TB  Is). 
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Table  5:  0TB  Data  Exclusion  Accounting 


Data  Pull/Exclusion 

Contracts  (C) 

Efforts  (E) 

OTBs 

0TB  Delta 

DAMIR/D  AES  (Nov  09) 

2267 

3231 

n/a 

n/a 

0TB  Initial  Data  Pull  (Nov  09) 

177 

220 

392 

n/a 

Exclusion  1 ;  Data  Purification 

177 

220 

318 

-74 

Exclusion  2:  Unstable  CBB 

71 

89 

143 

-175 

Exclusions:  Insufficient  Reports 

40 

47 

53 

-90 

Exclusion  4:  0TB  2s  removed 

40 

47 

47 

-6 

Final  0TB  Dataset 

40 

47 

47 

n/a 

Slope  Calculation 

Since  program  managers  typically  adjust  cumulative  CPI  to  a  level  or  value  of 
one  during  the  0TB  process  (Figure  6),  our  test  is  only  concerned  with  OTB's  effect  on 
the  cumulative  CPI  slope  before  and  after  0TB;  essentially,  we  are  looking  for  a  positive 
increase  in  the  cumulative  CPI  slope.  A  positive  slope  change  indicates  an  improvement 
in  management's  ability  to  earn  value  or  otherwise  stated,  management's  ability  to  reverse 
a  progressively  growing  overrun. 

We  utilized  simple  linear  regression,  method  of  least  squares,  to  calculate  the 
slopes  pre-  and  post-OTB.  With  the  cumulative  CPI  (continuous  variable)  on  y-axis  and 
Time  (in  months)  on  the  x-axis,  we  calculated  the  cumulative  CPI  slopes  for  all  47  0TB 
occurrences  with  the  following  Least  Squares  Line  equation  (McClave,  2008). 
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(1) 


Given  the  sum  of  errors  is  zero  and  the  sum  of  squared  errors  is  minimized, 
where  (beta  zero)  is  the  y-intereept  and  (beta  one)  is  the  slope  of  the  line 
(MeClave,  2008). 

Additionally,  using  equation  (1),  we  ealculated  eumulative  CPI  slopes  for  all  47 
OTBs  broken  out  by  programmatic  factors  of  appropriation  purpose  (RDT&E  or 
Procurement),  service  component  (Air  Force,  Army,  DoD,  Navy)  and  contract  type  (Cost 
Plus,  Fixed).  Further,  we  calculated  each  slope  by  factor  and  time  period  post-OTB. 
Varying  time  post-OTB,  we  utilized  six  post-OTB  timeframes  to  include  six,  nine, 
twelve,  eighteen,  twenty- four  and  all-months.  Consistent  with  DAU  guidance,  we 
utilized  all  pre-OTB  cumulative  CPI  data  points  to  calculate  the  cumulative  CPI  slope 
before  0TB  and  varied  time  after  0TB  starting  at  six  months  (typically  two  consecutive 
reporting  periods)  to  examine  the  time  effect.  These  categorical  and  time  breakouts 
enabled  further  sensitivity  analysis;  essentially,  we  tried  to  determine  if  OTB's  effect  on 
cumulative  CPI  post-OTB  was  sensitive  to  time  and/or  the  noted  programmatic  factors. 
Hypothesis  Test 

The  hypothesis  test,  performed  at  the  95%  confidence  level  (alpha  0.05), 
examines  OTB's  effect  on  cumulative  CPI  slope  before  (b)  and  after  (a)  0TB.  More 
specifically,  we  compare  the  median  {m)  location  of  two  population  probability 
distributions  relating  to  Cumulative  CPI  Slopes  Before  0TB  (mb)  and  Cumulative  CPI 
Slopes  After  0TB  (ma).  The  Null  Hypothesis,  Ho  (2),  states  that  mb  is  equal  to  ma.  The 
Alternative  Hypothesis,  Ha  (3),  states  that  mb  is  less  than  nia,  since  we  are  testing  for  a 
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positive  change  in  the  cumulative  CPI  slope  after  0TB,  it  is  a  one-tailed  test.  If  0TB  has 
an  effect  on  the  cumulative  CPI  median  slope  location,  we  will  reject  Ho  and  conclude 
that  Ha  is  true  (median  before  is  statistically  less  than  the  median  after).  If  0TB  has  no 
effect,  we  will  fail  to  reject  Ho  and  conclude  that  Ho  is  true  (median  locations  are 
statistically  equal). 


Null  Hypothesis  (Ho): 

mb  =  ma 

(2) 

Alternative  Hypothesis  (Ha): 

mb  <  ma 

(3) 

The  variable  of  interest,  cumulative  CPI  slope,  is  a  continuous  random  variable, 
not  normally  distributed  (Figures  7  and  8).  We  ran  Minitab®  Kolmogorov-Smirnov  (KS) 
test  to  determine  normality;  given  a  KS  value  greater  than  0.05  (alpha  level)  and  visual 
inspection  of  the  plotted  data,  we  reject  the  assumption  of  normality.  Given  normality 
fails,  we  must  employ  a  nonparametric  test  to  compare  the  median  locations. 
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Normality  Test  Cum  CPI  Slopes  Before  (b) 

Normal 


Mean 

0.01702 

StDev 

0.08918 

N 

47 

KS 

0.418 

P-Value 

<0.010 

Figure  7;  Normality  Test  for  Cumulative  CPI  Slopes  Before  0TB 


Normality  Test  Cum  CPI  Slopes  After  (a) 

Normal 


Mean 

-0.002426 

StDev 

0.007715 

N 

47 

KS 

0.239 

P-Value 

<0.010 

Figure  8:  Normality  Test  for  Cumulative  CPI  Slopes  After  0TB 
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Mann- Whitney  U-Test 

"Nonparametric  methods  (distribution- free  tests)  focus  on  the  location  of  the 
probability  distribution  of  the  population,  rather  than  on  specific  parameters  of  the 
population,  such  as  the  mean  (hence,  the  name  nonparametrics)"  (McClave,  2008). 
Specifically,  the  Mann- Whitney  U-Test  utilizes  a  research  design  involving  two 
independent  samples  that  represent  two  populations  with  different  median  locations 
(Sheskin,  2007).  In  our  research,  the  Mann- Whitney  U-Test  ranks  the  cumulative  CPI 
slopes  before  and  after  0TB  and  calculates  a  U-statistic  to  determine  if  there  is  a 
significant  difference  in  the  median  location  of  the  samples  tested. 

In  instances  where  the  Minitab®  Mann- Whitney  test  did  not  produce  a  p-value, 
the  IMP®  Wilcoxon  Rank  Sums  "2-Sample  Test,  Normal  Approximation,  Prob>|Z|" 
value  is  halved  to  calculate  a  one-tailed  p-value.  Concerning  the  Wilcoxon  Rank  Sum 
Test,  McClave  notes  "another  statistic  used  for  comparing  two  populations  based  on 
independent  random  samples  is  the  Mann- Whitney  U-statistic.  The  U-statistic  is  a  simple 
function  of  the  rank  sums.  It  can  be  shown  that  the  Wilcoxon  rank  sum  test  and  the 
Mann- Whitney  U-test  are  equivalent"  (McClave,  2008). 

The  following  assumptions  apply  to  the  use  of  the  Mann- Whitney  U-Test 
(Sheskin,  2008).  First,  the  samples  must  be  randomly  selected  from  the  population  they 
represent  (Sheskin,  2008);  in  2010,  Thickstun  found  that  the  occurrence  of  0TB  was  in 
fact  "random".  Our  research  pulls  these  random  occurrences  from  the  DAES  database 
via  DAMIR  retrieval. 

Further,  the  two  samples  must  be  independent.  Not  only  are  the  programs 
independent  (different  capabilities,  technical  risk,  schedule,  funding  and  management. 
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etc...),  the  slopes  before  and  after  0TB  are  independent  for  having  gone  through  the  0TB 
treatment;  "cumulative  indices  will  only  reflect  the  performance  since  the  new  baseline 
was  implemented"  (DAU,  2007). 

Beyond  independence,  the  variable  of  interest  must  be  a  continuous  random 
variable  (Sheskin,  2008).  Cumulative  CPI  slope  is  a  continuous  random  variable  in  that 
slope  can  take  on  any  value  and  this  variable  quality  mitigates  the  risk  of  ranking  ties. 

As  a  final  assumption,  Sheskin  notes  that  "the  underlying  distributions  from 
which  the  samples  are  derived  are  identical  in  shape"  (Sheskin,  2008).  Figure  9  depicts 
similarity  in  shape  with  the  exception  of  outliers;  the  Mann- Whitney  U-Test  adjusts  for 
outliers  in  comparison  to  other  parametric  tests. 


Probability  Plot  of  CPI  Slopes  b,  CPI  Slopes  a 

Normal  -  95%  Cl 


Variable 


CPI  Slopes  b 


— ■—  CPI  Slopes  a 


Mean  SCDev  N  AD  P 

0.01702  0.08918  47  11.748  <0.005 

•0.002426  0.007715  47  4.421  <0.005 


Figure  9:  Similar  Distribution  Shapes;  Cumulative  CPI  Slopes  Before,  After  0TB 
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Past  researchers  utilized  the  Mann- Whitney  U-Test  to  assess  the  effectiveness  of 
acquisition  reform  legislature.  Christensen  found  reform  efforts  from  1960  to  1999 
ineffective  in  reducing  average  cost  growth  of  20  percent  during  that  timeframe 
(Christensen,  1999).  In  2003,  Holbrook  "discovered  that  cost  performance  for  contracts 
completed  after  reform  initiative  implementation  was  no  different  than  cost  performance 
on  contracts  completed  before  implementation"  (Holbrook,  2003). 

Chapter  Summary 

In  Chapter  III,  we  explained  the  source  of  our  data,  provided  the  cumulative  CPI 
slope  calculation,  presented  the  hypothesis  test  and  discussed  the  assumptions  of  the 
Mann- Whitney  U-Test.  Having  established  the  validity  of  the  test  and  calculations 
performed,  we  present  the  results  of  our  statistical  tests  in  Chapter  IV  and  stress  our  data 
limitations.  In  Chapter  V,  we  draw  conclusions  based  on  our  results,  discuss  the  policy 
implications  of  our  findings  and  recommend  topics  of  further  study. 
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IV:  Results  and  Analysis 


In  Chapter  IV,  we  present  the  results  of  our  statistieal  tests,  analyze  the  outeomes 
and  diseuss  our  data  limitations.  In  Chapter  V,  we  eonelude  with  poliey  implieations  and 
reeommended  topies  of  further  study. 

Results 

To  summarize  the  hypothesis  and  tests  (95%  eonfidenee;  alpha  0.05),  utilizing 
Minitab®  Mann- Whitney  U-Test  and  JMP®  Wileoxon  Rank  Sums,  we  examine  the 
distribution  of  eumulative  CPI  slopes  before  and  after  0TB  for  a  ehange  in  median 
loeation.  Table  6  summarizes  the  P-Value  results  for  the  number  of  0TB  efforts  tested  in 
a  given  time  period  after  0TB  and  for  the  faetor  in  question.  Given  our  data  limitations, 
the  most  reliable  results  rest  in  the  "All"  row.  We  performed  further  sensitivity  analysis 
by  varying  time  after  0TB  by  factor,  but  the  results  are  limited  by  small  sample  sizes  (n) 
and  percentages  of  the  total  number  of  0TB  efforts. 

Concerning  table  interpretation  (Table  6),  P-Value  is  defined  as  "the  observed 
significance  level,  or  p-value,  for  a  specific  statistical  test  is  the  probability  (assuming  Ho 
is  true)  of  observing  a  value  of  the  test  statistic  that  is  at  least  as  contradictory  to  the  null 
hypothesis,  and  supportive  of  the  alternative  hypothesis,  as  the  actual  one  computed  from 
the  sample  data"  (McClave,  2008).  Essentially,  if  the  P-Value  is  less  than  alpha  (0.05), 
we  reject  Ho  and  accept  Ha;  if  the  P-Value  is  greater  than  alpha,  we  fail  to  reject  Ho. 
Considering  our  hypothesis  and  our  test  procedure,  failing  to  reject  Ho  means  that  0TB 
has  no  effect  on  changing  the  median  location  of  the  cumulative  CPI  slopes  before  and 
after  0TB;  the  median  locations  are  equal  (2).  Additionally,  the  Service  factor  of  "DoD" 
consistently  had  Insufficient  Data  Points  (IDP);  one  data  point  did  not  enable  testing. 
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Table  6;  Results  of  OTB's  Effect  on  Cumulative  CPI  Slope  After  0TB 


+6  mos 

n 

+9  mos 

n 

+12  mos 

n 

+18  mos 

n 

+24  mos 

n 

M 

n 

Cateogry 

RDTE 

0.0414 

8 

0.0667 

9 

0.0789 

9 

0.0927 

9 

0.0927 

9 

0.0927 

9 

Procurement 

0.0929 

10 

0.4885 

12 

0.4426 

12 

0.41995 

12 

0.4426 

12 

0.4655 

12 

Service 

Air  Force 

0.0867 

6 

0.0639 

8 

0.0946 

8 

0.1351 

8 

0.1592 

8 

0.1351 

8 

Army 

0.1345 

8 

0.056 

9 

0.0558 

9 

0.0666 

9 

0.0666 

9 

0.0666 

9 

DoD 

IDP 

1 

IDP 

1 

IDP 

1 

IDP 

1 

IDP 

1 

IDP 

1 

Navy 

0.4108 

27 

0.25165 

29 

0.2048 

29 

0.21605 

29 

0.24935 

29 

0.24935 

29 

Type 

Cost  Plus 

0.0547 

20 

0.0795 

22 

0.1025 

22 

0.1004 

22 

0.1004 

22 

0.1004 

22 

Fixed 

0.4026 

22 

0.3525 

25 

0.4155 

25 

0.4729 

25 

0.4884 

25 

0.4884 

25 

% 

AU 

0.1119 

42 

0.1389 

47 

0.1943 

47 

0.2092 

47 

0.2225 

47 

0.2092 

47 

Note:  P- Values  for  "n"  efforts  (Vary  Time  After  0TB  Only,  0TB  Is  Only,  Alpha  0.05,  Mann  Whitney  U  Test) 

Analysis 

With  the  exception  of  RDTE  at  +6  months,  all  p-values  (Table  6)  are  greater  than 
alpha  (0.05).  Given  our  results,  we  fail  to  reject  the  Ho  (2)  and  conclude  that  the  median 
location  of  the  cumulative  CPI  slope  distribution  before  0TB  is  equal  to  the  median 
location  of  the  cumulative  CPI  slope  distribution  after  0TB.  0TB  treatments  have  no 
effect  on  the  cumulative  CPI  slope  after  0TB.  Eurther,  OTB's  effect  on  the  cumulative 
CPI  slopes  after  0TB  is  not  sensitive  to  time  and/or  the  programmatic  factors  chosen  in 
this  study. 

Borderline  significance  is  noted  in  the  RDTE  category  at  +6  months  (p-value  of 
0.0414,  Table  6),  but  the  size  of  the  sample  (n  =  8)  and  limitations  of  the  data  do  not 
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allow  us  to  confidently  reject  the  Ho  and  state  that  0TB  has  an  effect  on  the  Cumulative 
CPI  slope  after  0TB.  Given  our  data  limitations  and  the  skepticism  that  surrounds  early 
CPI  index  use  after  0TB,  we  are  not  willing  to  commit  a  Type  I  error  and  incorrectly 
state  that  "0TB  increases  the  cumulative  CPI  slope  for  RDTE  contracts". 

At  the  90%  confidence  level  (alpha  0.10),  there  appears  to  be  significance  in  the 
rows  of  RDTE,  Army  and  Cost  Plus  contracts  (Table  6).  Given  data  limitations,  we  are 
unwilling  to  commit  a  Type  I  error  and  state  that  0TB  has  an  effect.  In  percentage  of 
total  sample,  RDTE,  Army  and  Cost  Plus  represent  19%,  19%  and  47%,  respectively. 
Again,  we  have  consciously  decided  to  risk  a  Type  II  error  vs.  a  Type  I  error  given  the 
small  percentages  and  relatively  high  p-values. 

Additionally,  when  comparing  "Service"  p-values  and  "Type"  contract  p-values 
(Table  6),  we  note  large  differences  between  the  row  factors.  Again,  any  significance 
that  could  potentially  be  drawn  Ifom  these  differences  is  diminished  by  small  sample  size 
and  percentages  at  the  factor  level  and  further  diminished  by  our  insignificant  finding  in 
"All". 

Limitations 

We  summarize  our  limitations  by  type  risk  of  statistical  error.  Concerning  the 
risk  of  a  Type  I  error  (risk  of  rejecting  a  true  null  hypothesis  or  “incorrectly  stating  there 
is  an  effect”),  the  most  obvious  limitation  is  our  small  sample  size  of  47  (Table  6). 

Within  our  table  of  results,  it  is  apparent  that  some  factor  and  time  intersections  have  a 
very  small  percentage  of  the  final  dataset,  the  largest  being  "Navy"  at  approximately 
62%.  The  majority  of  factor  percentages  are  less  than  26%  of  the  sample  total.  The  most 
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significant  row  in  Table  6  is  the  "All"  row;  it  represents  all  47  OTBs  being  tested  for 
treatment  effect  across  time  and  we  are  unable  to  reject  Ho  due  to  p-values  greater  than 
0.05  and  0.10. 

Additionally,  coneerning  the  risk  of  committing  a  Type  I  error,  we  note  the  small 
pereentage  of  0TB  contraet  efforts  being  studied  within  the  DAMIR  database  and  within 
the  population  of  system- identified  0TB  contract  efforts  (Table  5).  By  the  numbers, 
approximately  7%  of  contract  efforts  maintained  in  DAMIR  are  reported  as  0TB.  After 
necessary  exclusions  (Table  5),  our  47  0TB  contract  efforts  represent  approximately 
1.5%  of  total  DAMIR  contraet  efforts  and  21%  of  initially- identified  0TB  contract 
efforts. 

To  summarize  the  Type  I  limitations,  these  are  small,  purified  numbers  (Table  5) 
and  pereentages  that  support  the  validity  of  the  test  and  design;  we  should  have  seen  an 
effeet,  but  we  did  not  (Table  6).  We  are  unwilling  to  conelude  that  0TB  has  a  borderline 
effect  on  the  cumulative  CPI  after  0TB  and  risk  a  Type  I  error. 

Concerning  the  limitations  surrounding  a  Type  II  statistical  error  (risk  of  rejeeting 
a  true  alternative  hypothesis  or  “letting  an  effect  go  free”),  our  attention  turns  to  the  slope 
data  eontent.  In  our  researeh,  we  studied  cumulative  CPI  data.  The  cumulative  CPI  data 
is  historical  in  nature  and  based  on  “time  now”  totals  (EV  and  AC)  from  program 
inception.  This  quality  anehors  performance  to  the  past,  meaning  cumulative  CPI  is  very 
difficult  to  change.  Our  data  selection  increases  the  risk  of  a  Type  II  statistical  error, 
meaning  that  we  may  have  let  an  “effect  go  free”  by  choosing  averaged  performance 
data.  In  Chapter  V,  we  will  recommend  a  within-seope  data  remedy  to  address  this  issue. 
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V:  Discussion  and  Conclusion 


0TB  is  a  subset  of  cost  growth.  At  the  contract  effort  level,  the  stated  goal  of  an 
0TB  EVM  intervention  is  to  gain  managerial  control  of  a  project's  remaining  work;  for 
any  program,  in  any  status,  gaining  managerial  control  of  remaining  work  is  a  worthwhile 
goal.  The  0TB  process  is  a  bottoms-up,  collaborative  assessment  of  remaining  work  that 
is  centered  on  risk  analysis  in  which  cost  and  schedule  variances  are  typically  removed 
and  funding  is  added  in  excess  of  the  original  contract  budget  base.  The  0TB  guide 
states,  “it  is  important  that  the  project  managers  recognize  that  a  robust  risk  analysis  for 
the  remaining  project  has  resulted  in  a  realistic  schedule  and  budget  baseline. .  .it  is  now 
more  important  than  ever  to  have  a  risk  management  strategy  that  encompasses  integrated 
risk  analysis  and  risk  mitigation”  (DAU,  2007). 

Research  Questions,  Results  and  Limitations 

Our  research  studied  two  specific  questions  concerning  OTB's  effect  on 
cumulative  CPI  slope  after  0TB; 

1)  Does  the  0TB  process  (treatment)  improve  the  cumulative  CPI's  rate  of 
change  (cumulative  CPI  slope)  after  0TB? 

2)  Is  the  cumulative  CPI  slope  after  0TB  sensitive  to  time  and/or  programmatic 
factors  to  include  contract  type,  military  service  and  the  purpose  of  the  appropriation? 

For  the  data  studied,  we  find  there  is  no  statistically  significant  change  in 
cumulative  CPI  slope  after  an  0TB  intervention;  0TB  does  not  gain  managerial  cost 
control  of  remaining  work  with  respect  to  the  cumulative  CPI  slope.  Further,  we 
conducted  sensitivity  analysis  to  determine  if  time  and  programmatic  factors  affect  OTB's 
effect  on  cumulative  CPI.  We  find  borderline  significance  in  the  factors  of  RDTE,  Army 
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and  Cost  Plus  contracts,  but  given  the  data  limitations,  we  remain  unwilling  to  state  that 
0TB  has  an  effeet  on  gaining  managerial  control  of  cost  with  respect  to  the  cumulative 
CPI  slope.  We  conelude  that  0TB  does  not  inerease  the  eumulative  CPI  slope  after  0TB. 

Concerning  data  limitations,  in  order  to  validly  utilize  the  design  of  our  research 
and  perform  the  statistical  methods  discussed,  we  exeluded  a  large  percentage  of  0TB 
data  from  the  original  data  pull  (Table  5).  Having  utilized  only  21%  of  the  0TB  data  and 
its  small  percentage  of  the  total  eontract  efforts  maintained  in  DAMIR,  we  limit  our 
finding  that  0TB  has  no  effect  the  cumulative  CPI  slope  after  0TB.  Additionally 
eoneerning  results  presented  in  Table  6,  our  "borderline"  factor  significance  is  very 
unreliable  due  to  small  sample  sizes  and  small  percentages  of  the  total  0TB  contract 
efforts  studied.  Consistent  with  these  limitations,  we  avoid  Type  I  errors  by  failing  to 
reject  Ho  in  all  instances.  Finally,  our  choiee  to  study  cumulative  CPI  dampens  our 
ability  to  see  the  effect  of  current  management  actions  and  raises  our  probability  of 
making  a  Type  II  statistical  error;  essentially,  our  cumulative  data  selection  has  raised  our 
risk  of  incorrectly  stating  “no  effect.” 

Policy  Implications 

Qualified  by  our  limitations,  our  research  empirically  characterizes  0TB  as 
ineffective  in  improving  eost  performance  as  it  relates  to  improving  the  cumulative  CPI 
slope  after  0TB.  As  sueh,  we  reeommend  disallowing  the  implementation  of  a  formal 
0TB  unless  explicitly  justified  by  a  more  robust  and  standardized  0TB  cost/benefit 
analysis.  Per  DCMA’s  2006  EVM  implementation  guide,  once  the  eontractor  has 
submitted  an  0TB  request,  the  customer  has  30  days  to  approve  or  disapprove  the 
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request.  If  disapproved,  “the  PM  should  provide  speeifie  reasons  as  to  why  it  was  denied 
and  what  is  required  to  obtain  approval”;  PMs  should  utilize  this  approval  proeess  to 
require  a  more  stringent  eost/benefit  analysis  of  the  eontraetor  to  justify  the  0TB 
investment. 

From  the  eumulative  CPI  slope  viewpoint,  the  eontraetor  should  not  remove 
historieal  varianees,  add  funding  to  the  existing  requirement  and  eontinue  to  overrun  the 
program  at  the  same  eumulative  CPI  rate  pre-  and  post-OTB.  The  eumulative  CPI  slope, 
normally  a  negative  slope  that  denotes  a  progressively  growing  overrun,  should  improve 
for  having  gone  through  the  0TB  proeess.  At  a  minimum,  we  should  see  some  impaet 
within  the  first  six  months  after  0TB;  this  time  period  mitigates  the  eumulative  CPI  data 
anehoring  effeet.  Further,  the  eontraetor’s  justifieation  should  inelude  a  diseussion  of 
inereasing  the  eumulative  CPI  slope  after  0TB  and  the  differenee  in  estimated  overrun 
eosts  if  0TB  aetions  are  not  taken. 

Thiekstun  notes  that  0TB  eosts  are  in  addition  to  TAB;  essentially,  there 
is  an  ineremental  eost  of  doing  0TB  business  (Thiekstun,  2010).  That  ineremental  0TB 
eost  should  produee  a  quantifiable  return  on  investment.  The  eustomer  PM  should  be 
able  to  assess  the  “impaet  if  0TB  is  not  funded.”  We  believe  this  is  a  more  quantifiable 
way  of  aehieving  and  justifying  an  0TB  investment.  Cukr  states  it's  "possible  just  to 
eontinue"  (Cukr,  2000);  if  not  justified,  just  eontinue  and  save  the  time,  the  additional 
work,  the  historieal  information  and  the  finaneial  resourees. 

Future  Research 

Our  research  did  not  quantify  the  potential  eost  savings  of  such  a  policy  decision, 
but  future  research  should  attempt  to  determine  potential  savings. 
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In  terms  of  disallowing  0TB  completely,  we  recommend  gathering  additional 
evidence  of  “0TB  not  increasing  CPI  slope”  within  the  scope  of  this  study  based  on 
current  period  CPR  data  only,  not  cumulative.  This  approach  to  data  selection  will 
provide  greater  insight  into  current  or  more  near-term  PM  actions.  The  cumulative  data 
is  anchored  in  historical  performance,  meaning  it  is  very  difficult  to  change  and  limits  our 
finding  and  correspondingly,  the  policy  implication  of  disallowing  0TB.  However,  this 
limitation  does  not  eliminate  the  recommendation  to  better  justify  an  0TB  investment. 

Additionally,  the  acquisition  community  should  identify  “treatmenf’-type 
processes  in  the  acquisition  life  cycle,  perhaps  rolling  Integrated  Baseline  Reviews  (IBR) 
or  EVMS  surveillance  activities  tied  to  ANSI  guidelines,  to  implement  standardized, 
repeatable  assessments  at  the  contract  level.  Obtaining  this  data  in  a  systematic  and 
objective  nature  will  allow  the  acquisition  community  to  research  meaningful 
relationships  between  program  actions  and  superior  contract  performance.  Once 
identified,  program  managers  can  build  their  execution  plans  centered  on  the  most 
effective  actions  to  effect  improved  cost,  schedule  and  performance  outcomes. 
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